Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[IOTDB-6328] Add optimization for aggregation query in align by device with template situation #12513

Merged
merged 35 commits into from
May 28, 2024
Merged

[IOTDB-6328] Add optimization for aggregation query in align by device with template situation #12513

merged 35 commits into from
May 28, 2024

Conversation

Beyyes
Copy link
Member

@Beyyes Beyyes commented May 11, 2024 •
edited
Loading

Description

This pull request is a follow up optimization of #11783.

When all queried devices is set in one device template, we don't need to serialize the related variables as the count of devices, we just need extracted the common variables and serialize and deserialize them for once.

The experiments result is in this link: https://apache-iotdb.feishu.cn/docx/Pp7sdOucCoMLNTxyA9dce6OXnLe

@Beyyes Beyyes marked this pull request as draft May 11, 2024 12:58
@codecov Codecov
Copy link

codecov bot commented May 11, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 6.27675% with 657 lines in your changes are missing coverage. Please review.

Project coverage is 42.97%. Comparing base (cb1bca6) to head (b6acca4).
Report is 1 commits behind head on master.

Files Patch % Lines
...queryengine/plan/planner/TemplatedLogicalPlan.java 0.00% 143 Missing ⚠️
...gine/plan/analyze/TemplatedAggregationAnalyze.java 0.00% 114 Missing ⚠️
...tdb/db/queryengine/plan/analyze/TemplatedInfo.java 0.00% 87 Missing ⚠️
...ueryengine/plan/planner/OperatorTreeGenerator.java 9.52% 76 Missing ⚠️
...yengine/plan/optimization/AggregationPushDown.java 18.39% 71 Missing ⚠️
.../db/queryengine/plan/analyze/TemplatedAnalyze.java 3.77% 51 Missing ⚠️
.../node/source/AlignedSeriesAggregationScanNode.java 3.22% 30 Missing ⚠️
...ine/plan/planner/plan/node/process/FilterNode.java 17.85% 23 Missing ⚠️
...plan/planner/plan/node/process/DeviceViewNode.java 0.00% 21 Missing ⚠️
...gine/plan/planner/TemplatedLogicalPlanBuilder.java 0.00% 17 Missing ⚠️
... and 10 more
Additional details and impacted files 
@@             Coverage Diff              @@
##             master   #12513      +/-   ##
============================================
- Coverage     43.09%   42.97%   -0.12%     
  Complexity       69       69              
============================================
  Files          3190     3191       +1     
  Lines        198299   198847     +548     
  Branches      23613    23707      +94     
============================================
+ Hits          85459    85460       +1     
- Misses       112840   113387     +547

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Beyyes and others added 24 commits May 13, 2024 08:35
@Beyyes
fix sourcenode
8699ede
@Beyyes
fix template serizlize error; fix filter can not push down bug
e7985fa
@liuminghui233
process ProjectNode
1592507
@liuminghui233
fix UT
8ba6959
@liuminghui233
add more UT
d1924f3
@liuminghui233
fix IT
41f0b02
@Beyyes
not support agg(*) or agg(s1+1) now; perfect serialize method of RawD…
884dcca 
…ataAggregation
@liuminghui233
fix extend aligned path
613a344
@Beyyes
merge master
6a69442
@Beyyes
fix ut
303df4b
@Beyyes
Merge branch 'lmh/fixPushDownProject' into beyyes/agg_template_alignb…
a70fc78 
…ydevice
@Beyyes
fix descending aggregator problem; add more comments
c8199d2
@Beyyes
fix group by time; fix select agg1() having agg2()
c1b9f3a
@Beyyes
fix typeProvider.getTemplatedInfo().groupByTimeParameter problem
bd76f67
@Beyyes
merge with master
135c808
@Beyyes
add IoTDBAlignByDeviceWithTemplateAggregationIT
7d8f8f8
@Beyyes
support sliding window, fix visitSlidingWindow method of OperatorGene…
8854da1 
…ratorTree; support order by time; add schemaMap for templateInfo; fix devicePath of SingleDeviceViewNode.
@Beyyes
add ascending and descending aggregation descriptors support
569d6ed
@Beyyes
merge master
3d42a2a
@Beyyes
fix merge error in TemplatedAnalyze
ef77dc5
@Beyyes
Merge branch 'master' into beyyes/agg_template_alignbydevice
d16c17e
@Beyyes
fix outputEndTime serialize in templateInfo. remove redundant judgeme…
2a1b868 
…nts in ExpressionTypeAnalyzer
@Beyyes
add count_time support, add more its
126f856
@Beyyes
fix select agg where s1>1 bug which is aggreagtion pushdown situation
d86728d
@Beyyes Beyyes marked this pull request as ready for review May 28, 2024 06:05
@Beyyes Beyyes changed the title Add aggregation template align by device optimization Add optimization for aggregation query in align by device with template situation May 28, 2024
@Beyyes Beyyes changed the title Add optimization for aggregation query in align by device with template situation [IOTDB-6328] Add optimization for aggregation query in align by device with template situation May 28, 2024
@sonarcloud SonarCloud
Copy link

sonarcloud bot commented May 28, 2024

Quality Gate Failed Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 80%)
4.1% Duplication on New Code (required ≤ 3%)

See analysis details on SonarCloud

@Beyyes Beyyes merged commit 1af2c23 into master May 28, 2024
57 of 59 checks passed
@Beyyes Beyyes deleted the beyyes/agg_template_alignbydevice branch May 28, 2024 14:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@JackieTien97 JackieTien97 JackieTien97 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@Beyyes @JackieTien97 @liuminghui233